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introduction 

The availability of new technologies for the cock¬ 
pit and air traffic control facilities is creating new 
capabilities for enhanced aircraft operations and, 
with them, the need to evaluate the effectiveness of 
these new technologies in operational settings. Two 
such systems, Cockpit Display of Traffic Informa¬ 
tion (CDTI) and Automatic Dependent Surveillance 
- Broadcast (ADS-B), were recently demonstrated in 
an operational evaluation (OpEval) conducted at the 
Airborne Express Airpark in Wilmington, Ohio. The 
OpEval was sponsored by the Cargo Airlines Associa¬ 
tion (CAA) and the SafeFlight21 Office of the Fed¬ 
eral Aviation Administration (FAA), and included 
aircraft and flight crews from industry, government, 
and private organizations. 

The purpose of the OpEval was to demonstrate the 
use and expected benefits of CDTI, which consist of 
increased safety, efficiency, and capacity. Some spe¬ 
cific examples of these benefits are: enhanced visual 
acquisition for “see and avoid,” enhanced visual 
approaches, and efficient departure and final ap¬ 
proach spacing. An important part of this evaluation, 
from an ATC human factors perspective, was the 


analysis of objective flight data to quantify and con¬ 
firm the demonstrated benefits of the new technolo¬ 
gies. However, because of the complex nature of the 
OpEval, innovative field assessment techniques had 
to be developed and conducted to accomplish that 
purpose. This paper describes the development of these 
assessment procedures and the resulting analysis. 

METHOD 

The OpEval was conducted on July 10, 1999, at 
Airborne Express Airpark in Wilmington, Ohio. 
Thirteen aircraft of different types (primarily B727’s 
and DC-9’s) were involved in flying multiple traffic 
patterns using two parallel runways, 22L and 22R 
(see Figure 1). Flights took place between 0900-1100 
hours and between 1300-1600 hours (local time). 
Each pattern for each aircraft was assigned to either 
the CDTI or baseline (no CDTI) condition, accord¬ 
ing to the experimental design. A total of 168 pat¬ 
terns were flown throughout the day. Ail aircraft were 
in radio contact with Dayton TRACON and 
Wilmington Tower. Air traffic controllers from both 
facilities provided instructions and clearances as 
needed. 



1 





































Both subjective and objective data were recorded 
as part of the OpEval. Human factors observers were 
stationed on the flight decks of participating aircraft 
and collected various types of data, including crew 
response times to air traffic control (ATC) commu¬ 
nications and crew interaction with CDTI systems. 
Observers were also present in the Dayton TRACON 
to assess the effects of CDTI use on subjective con¬ 
troller workload. ATC system data were routinely 
recorded by the control facilities involved: Radar and 
voice communication data were recorded by Dayton 
TRACON, and tower voice communications were re¬ 
corded by Wilmington Tower. Indianapolis Air Route 
Traffic Control Center recorded radar data for activities 
occurring outside the TRACON’s airspace. These data 
were made available to researchers for analysis. 

Data Analysis 

An important objective of this analysis of ATC 
data was to develop databases of aircraft locations (as 
indicated by radar), as well as important characteris¬ 
tics of each traffic pattern flown, such as when events 
occurred (e.g., crossing the runway threshold on 
completion of a pattern). These databases were used 
as input for other analyses, such as the measurement 
of aircraft pair distances at the time of approach 
clearances. 

Another objective of this analysis was to obtain 
measures of accuracy and variance for CDTI-based 
spacing during the stage of flight known as the final 
approach. Comparison of these measures with those 
collected during the baseline condition can provide 
evidence as to the effectiveness of CDTI use. Ac¬ 
complishing this objective required the identifica¬ 
tion of discrete spacing events, including specific 
times and aircraft pairs. Additionally, the distances 
between aircraft during the events had to be calcu¬ 
lated. Achieving both of the analysis objectives, data¬ 
base construction and final approach spacing, required 
the development of a series of procedures, described 
in detail below. 

Three primary data sources provided the basis for 
analyses. These included ATC radar system record¬ 
ings provided by Dayton TRACON, transcripts of 
all voice communications between the TRACON 
and participating aircraft (in the form of electronic 
spreadsheets), and records that indicated under which 


condition—CDTI or baseline—each aircraft flew on 
each pattern. Information from these sources was 
used to accomplish the objectives of the analysis, as 
described in the steps below. 

Step 1: ATC Reduction of Radar Data. 

The radar system recordings provided by Dayton 
TRACON had to be further processed by an FAA 
National Airspace System software program, “CDR 
Editor,” on an Automated Radar Terminal (ARTS) 
mainframe computer. Because the volume of data for 
this analysis was so great, it was not practical to 
process the data at the Dayton facility. Such process¬ 
ing requires the allocation of significant memory and 
processing resources on the ARTS computer; because 
TRACON facility computers continuously process 
live traffic data, it was not possible to use one of them 
for such a task. Consequently, the OpEval data were 
processed by the ARTS computer located at the FAA 
Academy in Oklahoma City. This system is routinely 
used for training and instructional purposes. 

The CDR Editor produced approximately 4.3 
million lines of textual output, consisting mainly of 
aircraft position reports and ATC system messages. 
The output was transferred to a Windows NT-based 
computer network, upon which subsequent analyses 
were performed (See Figure 2). The Windows-based 
computer platform allowed maximum flexibility for 
researchers to use application software such as spread¬ 
sheet and database programs and to develop the 
customized software (written in Visual Basic) that 
performed much of the subsequent analysis. The 
CDR Editor output was also transferred to a Unix- 
based computer network for use with the Systematic Air 
Traffic Operations Research Initiative (SATORI) soft¬ 
ware, designed to utilize TRACON data. (Rogers, 1993). 

Step 2: Development of Aircraft Location and 
Pattern Events Databases. 

To perform analyses involving aircraft positions, 
databases were needed that contained information 
about the recorded locations of all participating air¬ 
craft throughout the OpEval. In addition, a database 
was needed that defined each pattern flown by each 
aircraft and the important events within each pattern. 
Once developed, these databases served as the basis 
for further analyses, as well as for archival purposes. 
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Radar Track Records for Relevant Aircraft. To de¬ 
velop the database of aircraft locations, all radar 
tracking records for the participating aircraft had to 
be identified. These records were produced each time 
the ATC radar sweep made contact with an aircraft. 
The CDR Editor output data consisted of all 
TRACON system messages and radar tracking re¬ 
ports recorded from two radar sites (Dayton and 
Wilmington) during an eight-hour period that en¬ 
compassed the OpEval. Radar tracking reports com¬ 
prise several data fields, including time of day, aircraft 
identification code (ACID), assigned beacon code 
(ABC), reported beacon code (RBC; reported by the 
aircraft’s transponder), and x,y coordinates, relative 
to the radar site (see Figure 2). 

Software was developed that extracted tracking 
records for which the ACID matched one of the 
participating aircraft in the OpEval. However, in¬ 
spection of the results of this procedure revealed that 
no records existed for participating aircraft at alti¬ 
tudes below 1500 feet (mean sea level), even though 
the field elevation at the airpark was approximately 
1100 feet. Further examination showed that tracking 


records did exist for aircraft at altitudes below 1500 
feet, although the ACID data fields were blank. 
These records did, however, have values in the ABC 
field, which could also be used to identify aircraft. 

To identify an aircraft by the ABC, it is necessary 
to know which beacon code had been assigned to that 
aircraft at the time of the data record. Knowing the 
specific time is necessary because the beacon code 
assignment for a particular aircraft can be changed 
during the day (see Figure 3 for examples of these 
characteristics of the CDR editor output). Therefore, 
software was written that scanned the CDR Editor 
output and determined which beacon codes were 
assigned to which aircraft at specific times. This was 
done by sorting the records that had values in the 
ACID field by time and then determining the periods 
during which each ACID was assigned to specific 
beacon codes. Once this was accomplished, other 
software extracted all records that corresponded to 
participating aircraft by matching either on ACID or 
on ABC. This process resulted in a complete data¬ 
base, coded by ACID, of tracking records for all 
aircraft involved in the OpEval. 


TRACKING DATA 


7/10/99 


PAGE 

71500 




TIME 

ACID 


ABC 

RBC 

FRM 

RALT 

X 

Y 

HDG 

SPD ... 

19:56:16.445 


255 

1200 

1200 

38 

3500 

-13.69 

-19.19 

266 

90 ... 

19:56:16.445 


112 

1200 

1200 

19 

2400 

11.25 

46.81 

256 

57 ... 

19:56:16.446 

FDX9002 

143 

4515 

4515 

30 

2600 

-1.06 

-1.25 

191 

188 ... 

19:56:16.447 

UAL1288 

18 

1364 

1364 

38 

8600 

5.38 

17.38 

175 

294 ... 


Figure 2. CDR Editor Output - Tracking Records 


TRACKING DATA 


7/10/99 

TIME 

ACID 


ABC 

RALT 

10:45:43.978 

FDX9002 


;4276 

1800 

10:45:48.474 

FDX9002 


4276 

1700 

14:07:39.335 

FDX9002 

•» • 

4204 

1500 

14:07:43.731 

FDX9002 


42U4 

1500 

14:07:48.123 


, • 

4204 

1400 

14:07:52.618 



4204 

14 00 

14:08:23.380 


;. : i . • 

4204 

1300 

14:08:27.680 

FdJc9002 


4204 

1500 

18:11:42.500 

OPS606 

: : 


2900 

18:11:47.087 

UPp606 

1 * 

4276 

2900 


| Aircraft has 
I different assigned 
I beacon codes at 
I different times. 


Aircraft Ip is not 
present when 
altitude < 1500. 


Same beacon code is assigned 
to different aircraft at ^ 
different times. 


Figure a CDR Editor Output - Aircraft Identification 
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Development of the Pattern Events Database . Many 
of the analyses planned for the OpEval required the 
development of a database that described key events 
and characteristics of each pattern flown. Examples 
of these include the times the aircraft began and 
completed the pattern, the pattern altitude, and the 
experimental condition (CDTI or baseline). The 
determination of these values was accomplished in a 
variety of ways as described below^ As events and 
characteristics were identified, the related informa¬ 
tion was entered into a database either electronically 
or manually, depending on the method of identifica¬ 
tion (See Figure 4). 

Several events (Departure, Downwind Leg, and 
Start of Descent) were identified by scanning the 
altitude profile of each aircraft. To do this, the 
database of tracking records was sorted by ACID and 
then by time of day. The beginning of the Departure 
stage for each pattern was obtained by scanning these 
records. The end of the Departure stage and the 
beginning of the Downwind Leg were similarly de¬ 
termined and defined as the point at which the 
aircraft reached pattern altitude (for that pattern, 
altitudes were assigned by ATC and were sometimes 
changed from one pattern to the next). Start of 
Descent was identified as the point at which the 
aircraft began descending from pattern altitude. 


Approach clearance times were manually extracted 
from the voice communications transcripts. This was 
accomplished by searching for keywords such as 
“cleared,” as in the ATC message, . .CLEARED ILS 
TWO TWO RIGHT APPROACH.” Another key 
event was the time at which the aircraft passed the 
outer marker. The outer marker is a navigational aid 
that aircraft typically cross after they have been cleared 
for runway approach. Outer marker crossing times 
were determined by software that tracked the aircraft’s 
locations (using x,y coordinates) from the time of the 
approach clearance to the time it crossed the coordi¬ 
nate location of the outer marker for the appropriate 
runway. Similarly, runway threshold crossing time 
was determined by tracking the aircraft from the time 
of crossing the outer marker to the coordinate loca¬ 
tion of the threshold of the correct runway. Figure 5 
illustrates several of the these events for a hypotheti¬ 
cal flight pattern. 

Step 3: Identification of Approach Events. 

To measure the spacing performance of aircraft 
during the final approach stage of flights, it was 
necessary to identify when specific aircraft pairs were 
both on final approach and lined up with the same 
runway. These periods will be called approach events . 
An approach event was identified for a trailing and 


Pattern ACID Runway Condtn 

Dep Level 

Ptrn Alt 

Clear Marker 

Threshold ... 

1 FDX9001 22L 

baseline 

13:27:17 13:29:19 

4000 

13:36:23 13:39:30 

13:41:07 \... 

2 FDX9001 22L 

CDTI 

13:41:07 13:43:13 

4000 

13:48:34 13:51:59 

13:53:45 ... 

1 ABX33 22R 

CDTI 

\ 

18:03:58 18:05:34 

3000 

18:08:57 18:68:59 

18:09:04 I... 

2ABX33 22R 

CDTI 

18:12:07 18:13:11 

3000 

18:18:09 j 18:20:23 

18:22:48 L. 


Figure 4. Pattern Events Database 



Figure 5. Stages of Flight 
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leading aircraft pair from the time both were aligned 
with the same runway until the lead aircraft crossed 
the runway threshold, (see Figure 6.) 

The identification of the OpEval approach events 
was accomplished with the use of the TRACON 
SATORI system. SATORI presents a graphical re¬ 
creation of TRACON radar information, including 
the synchronized replay of voice communications. 
This system allowed researchers to review aircraft 
movements and identify approach events. The iden¬ 
tity of the aircraft pairs plus the start and end times 
of each event were then entered into a database for 
further analyses. 


Approach Event has not started. 



Approach Event 



Approach Event ends when lead aircraft 
crosses runway threshold. 

Figure 6 . Approach Event Identification. 



Aircraft 1: Position 
recorded at 00:00:00 

Aircraft 2: Position 
recorded at 00:00:01 


Figure 7. Radar Aircraft Position Recordings 


Step 4: Calculation of Aircraft Locations and 
Distances. 

To measure the accuracy of aircraft spacing, the 
distance between each trailing and lead aircraft dur¬ 
ing each approach event had to be calculated. 

Therefore, the coordinate locations of each air¬ 
craft at specific times during the event had to be 
obtained. The radar antenna at the Dayton TRACON 
sweeps the airspace in a circular direction approxi¬ 
mately every five seconds, resulting in a position 
report for each aircraft each five seconds. Figure 7 
illustrates how this tracking procedure results in 
position reports at different times for different air¬ 
craft during an individual sweep. Therefore, direct 
distance calculations from the tracking records are 
not possible. To address this situation, software was 
developed that uses linear interpolation to obtain the 
coordinate positions of aircraft at specific times. 
These positions were then used to calculate distances 
between aircraft during each approach event. 

Step 5: Production of Graphical Profiles. 

To enable a better understanding of OpEval air¬ 
craft dynamics, the TRACON Radar Charting Sys¬ 
tem (TRACS) was developed. TRACS is a software 
system that allows researchers to view graphical radar 
profiles of flights that occurred during a specified 
time interval, such as the OpEval. The program 
presents the movements of data blocks on a screen. 
The textual information in the blocks indicate air¬ 
craft identity and altitude, and the position of the 
blocks indicates lateral position. It is also capable of 
displaying symbols that represent the histories of the 
movement of the aircraft. 

TRACS can also use information from the Pattern 
Events database to graphically code these histories to 
indicate the occurrence of key events, such as ap¬ 
proach clearances, or to indicate characteristics of 
individual patterns, such as the experimental condi¬ 
tion in effect. In addition, the screen output from 
TRACS can be printed to provide permanent records 
of these profiles. 
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RESULTS 

Many of the results derived from the analyses of 
tracking data are reported and discussed in the OpEval 
Final Report (Operational Evaluation Coordination 
Group, in press). 

Several results that emerged specifically from the 
analysis of the tracking data are presented here. A 
comprehensive review and discussion of the OpEval 
findings are available in the referenced report. Be¬ 
cause of the demonstrational nature of the OpEval, 
inferential statistical analyses were not conducted on 
these data. However, analyses involving descriptive 
statistics may provide some insight into the effects of 
CDTI use during the OpEval, and are therefore 
reported below. 

Approach Event Occurrence 

A total of 88 approach events occurred during the 
OpEval. Of these, 47 occurred in the morning, and 
41 occurred in the afternoon. The duration of the 
events ranged from a minimum of 22 s to a maximum 
of 160 s (See Table 1). 

Approach Event Spacing 

Because of low visibility conditions in the morn¬ 
ing, those approaches were flown using instrument 
landing systems (ILS), as opposed to the afternoon, 
during which all approaches were visual. Because of 
this significant difference in flying conditions, spac¬ 
ing measurements are presented separately for morn¬ 
ing and afternoon approaches. Table 2 includes means 
and standard deviations for approach event spacing. 


Table 1 . Approach Event Occurrence 
and Durations. 


Approach 

Event 

Number of 
Approach 
Events 

Mean 

Duration of 
Events 
(Seconds) 

Morning 

Baseline 

18 

82.9 

CDTI 

29 

81.1 

Afternoon 

Baseline 

6 

72.5 

CDTI 

35 

65.8 


Table 2. Approach Event Spacing. 


Approach 

Event 

Mean 

(nautical miles) 

Standard 

Deviation 

Morning 

Baseline 

7.3 

1.2 

CDTI 

6.8 

1.8 

Afternoon 

Baseline 

4.6 

1.2 

CDTI 

3.6 

1.4 


Figures 8 and 9 depict the frequency distributions of 
spacing distances for the baseline and CDTI patterns, 
flown in the morning and afternoon, respectively. 

For a number of reasons it is difficult to form 
conclusions about differences in spacing performance 
between the baseline and CDTI conditions. The 
required use of ILS in the morning probably reduced 
the opportunity for flight crews to use CDTI, espe¬ 
cially during the last portion of final approach, when 
spacing was determined by ATC instructions and the 


Spacing Distance - AM 



B Baseline 
□ CDTI 


Figure 8 . Frequency distribution of spacing measurements for the AM session. 
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Figure 9. Frequency distribution of spacing measurements for the PM session. 



Figure 10. TRACS output—all morning flights. 
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use of the ILS. During the afternoon, comparisons 
were also problematic because of the small number (6) 
of baseline approach events, compared with CDTI 
approach events (35). Nevertheless, the distributions 
seem to indicate reduced spacing for the CDTI condi¬ 
tions during both morning and afternoon. The smaller 
spacing distances observed for the afternoon patterns, 
compared with those in the morning, are expected 
because crews were able to perform visual approaches. 

Graphical Flight Profiles 

Figures 10 and 11 depict output samples from the 
TRACS program. Figure 10 illustrates all morning 
OpEval flights, coded by baseline or CDTI condi¬ 
tion. Each small circle on the display represents a 


tracked aircraft position. Each larger circle indicates 
an aircraft’s position at the time it was issued an 
approach clearance by ATC. 

Figure 10 reveals that in general, approach clear¬ 
ances were issued for runway 22R (top runway in 
figure) earlier in the pattern than those issued for 
runway 22L. It also appears that more aircraft flew in 
the CDTI condition on runway 22L, and more flew 
in the baseline condition on runway 22R. Both of 
these observations are examples of the types of infor¬ 
mation that could be valuable during the course of 
data analysis. 

Figure 11 represents all afternoon flights, also 
coded by condition. Comparison of the afternoon 
flight profiles with those from the morning illustrates 



Figure 11. TRACS output - All afternoon flights. 
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an interesting characteristic of the OpEval—that in 
general, the shapes of patterns flown in the morning 
conformed more to a “typical” flight pattern, such as 
that depicted in Figure 5, than did the patterns flown 
in the afternoon. Afternoon pattern shapes were 
markedly more variable than those from the morn¬ 
ing, perhaps due to differences in ATC instructions 
associated with visual approaches. 

CONCLUSION 

The introduction of new technologies in aviation, 
such as improved communication, navigation, and 
surveillance systems, is leading to an evolution of the 
complex interaction between aircrews and ATC. 
While these technologies offer significant benefits to 
users of the aviation system, their applications must 
be evaluated operationally prior to widespread 


deployment. Programs such as the demonstration 
conducted in Wilmington will become increasingly 
important, as will the analyses of objective flight data 
from such operations. The evaluation of the large 
volume of data generated by such complex demon¬ 
strations is challenging, but it can be accomplished 
by combining modern software tools with the devel¬ 
opment of new procedures and analysis techniques. 

REFERENCES 

Operational Evaluation Coordination Group (2000). 
CAA/FAA ADS-B / Safeflight 21 Phase I- Opera¬ 
tional Evaluation Final Report, (in press). 

Rogers, M.D., & Duke, D.A. (1993). SATORI: 
Situation Assessment Through Re-creation of 
Incidents. The Journal of Air Traffic Control , 
33(4), 10-14. 


9 



